“A Distorted Skull Lies in the Bottom Center...” Identifying Paintings from Text Descriptions

Most question answering systems use symbolic or text information. We present a dataset for a task that requires understanding descriptions of visual themes and their layout: identifying paintings from their descriptions. We annotate paintings with contour data, align regions with entity mentions from an ontology, and associate image regions with text spans from descriptions. A simple embedding-based method applied to text-to-image coreferences achieves state-of-the-art results on our task when paired with bipartite matching. The task is made all the more difficult by scarcity of training data. 1 Knowledge from Images Question answering is a standard NLP task that typically requires gathering information from knowledge sources such as raw text, ontologies, and databases. Recently, vision and language have been amalgamated into an exciting and difficult task: using images to ask or answer questions. While humans can easily answer complex questions using knowledge gleaned from images, visual question answering (VQA) is difficult for computers. Humans excel at this task because they abstract key concepts away from the minutiae of visual representations, but computers often fail to synthesise prior knowledge with confusing visual representations. We present a new instance of visual question answering: can a computer identify an artistic work given only a textual description? Our dataset contains images of paintings, tapestries, and sculptures covering centuries of artistic movements from dozens of countries. Since these images are of cultural importance, we have access to many redundant descriptions of the same works, allowing us to create a naturalistic but inexpensive dataset. Due to the complex and oblique nature of questions about paintings, their visual complexity, and the relatively small data size, prior approaches used for VQA over natural images are infeasible for our task. We formalise the task in Section 3, where we also present a preliminary system (ARTMATCH) and compare with it a data-driven text baseline to illustrate the usefulness and versatility of our method (Section 4). Finally, in Section 5 we compare our task and system to previous work that combines NLP and vision.